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I. Introduction 

In November 2002, NASA revised its Integrated 
Space Transportation Plan (ISTP) to evolve the Space 
Launch Initiative (SLI) to serve as a theme for two 
emerging programs. The first of these, the Orbital 
Space Plane (OSP), was intended to provide crew- 
escape and crew-transfer functions for the ISS. The 
second, the NGLT Program, developed technologies 
needed for safe, routine space access for scientific 
exploration, commerce, and national defense. 

The NGLT Program was comprised of 12 projects, 
ranging from fundamental hi gh- temperature materials 
research to frill-scale engine system developments 
(turbine and rocket) to scramjet flight test The 
Program included technology advancement activities 
with a broad range of objectives, ultimate 
applications/timeframes, and technology maturity 
levels. An over-arching Systems Engineering and 
Analysis (SE&A) approach was employed to focus 
technology advancements according to a common set 
of requirements. Investments were categorized into 
three “segments” of technology maturation: 
propulsion technologies, launch systems 
technologies, and SE&A. 

At the time of cancellation, the Program was 
pursuing the following major technology thrusts: 

• Development of a reusable liquid- 
oxygen/liquid-kerosene rocket booster 
engine. 

• Development of hypersonic, air-breathing 
propulsion and airframe systems. 

• Development of cross-cutting vehicle system 
technologies, intended to support a broad 
variety of launch and flight vehicle 
architectures. 


• Systems analysis activities to guide program 
investment and to ensure an appropriate fit 
with both NASA and Department of Defense 
(DoD) needs. 

The NGLT Program was intended to bring an array 
of technologies to a state of readiness appropriate to 
facilitate decisions near the end of the decade on 
whether or not to initiate a program for development 
of NASA’s next generation of launch vehicle(s). The 
resulting new program would then be built, in large 
part, on the selected technologies from NGLT. In 
short, NGLT was NASA’s investment in new space 
launch technologies for use primarily beyond the 
OSP. 

The team charged with collecting and processing 
NGLT lessons learned began with a call to Program 
personnel. Subgroups of the team reduced an initial 
set of 315 inputs by assessment and combination to 
100; a second and third cycle of assessment reduced 
those to a core set of 68. 

II. Key Findings 

Of the total number of lessons learned, the following 
were considered to be the “key findings” from each 
theme. (These are cross-referenced to supporting 
lessons learned (unique identifier in parentheses) in 
Section 3.0.) 


A. Requirements 

• Systems analysis should be used to guide and 
assess technology development, beginning 
early in the program. Systems analysis should 
develop quantifiable priorities and 
requirements against which technology 
projects execute (Rl, R2). 
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# Programs should establish reference mission 
and vehicies/requirements and then implement 
standard, well-documented flow-down 
practices to all technology activities, properly 
educating all personnel regarding those 
practices (R2, R3). 

# Programs should carefully assess the impacts 
and relevancy of legacy projects toward 
program goals and objectives before 
committing resources (R4). 

B. Plan 

• Programs should define roles and 
responsibilities early and all the way down to 
the supplier level, and communicate and 
enforce established protocols, responsibilities, 
and team core values early and often so that 
expected standards of operation are established 
for the life of the program (PI). 

• The formulation process should not be 
shortchanged — many critical issues will 
surface that will later drive cost and schedule 
(P2,P3,P4,P5). 

• Programs should perform a thorough 
assessment of the manufacturing vendor 
capability required to produce space-qualified 
hardware for each development spiral (P6). 

C. Schedule 

• Adequate planning should be performed early 
and realistic schedules developed with both 
sufficient and visible margins. This should be 
done initially from as low a level as possible 
and integrated in coordination with functional 
leads and analysts (SI, S3, S4). 

• An integrated master schedule should be 
developed early in the formulation phase, 
which can evolve to accommodate the 
maturity of the program (S2). 

D. Budget 

♦ Credible, independent cost estimates are 
required early in the project life-cycle (Bl, 
T8). 

♦ Initial budget estimates must include budget 
reserves and schedule margins for each major 
period of performance, with those reserves 
and margins assigned in proportion to the risk 
in each period. (B2). 

* Program management should establish a 
centralized resource-integration function 


early; the resulting team should have the 
capability for program-wide budget planning, 
analysis, tracking, reporting, and change 
control (B3). 

• Budget plans should take into account “real 
world” budget issues, such as delays and 
disbursements, continuing resolutions, and 
current limitations in the Integrated Financial 
Management (IFM) System (B4, B5). 

EL Agreements 

• Inter-Agency coordination of technology 
development should be pursued at the 
performing level (versus high-level, broad- 
based agreements), capitalizing on hardware 
developed and unique capabilities of the 
partner agencies (Al, A2). 

• Partnering agreements should be 
comprehensive. These should include 
resources (including reserves and procedures 
for addressing cost growth) and Safety and 
Mission Assurance (S&MA), as well as a 
clear definition of the roles and 
responsibilities of each team member (A3, 
A4, A5). 

• International relations require constant 
attention. The NASA international 
agreements process should be streamlined. 
Programs should leverage existing 
international agreements to the greatest extent 
possible (A6, A7). 

• When Centers perform work for contractors 
(e.g., through Government Task Agreements), 
the contractor must be given full 
responsibility and accountability for managing 
resources (A8). 

• Programs should ensure that contracts allow 
for effective utilization of both government 
and industry resources: ask only for what the 
government can effectively use — no more — 
(A9, A10) — and enable flexibility in the 
execution of future work (All, A12, A13, 
A14). 

F. Execution 

• NASA engineering staffs should be utilized to 
provide technical insight/independent analysis 
in high-risk, mission-critical areas. NASA 
should plan and budget for the appropriate 
level of insight early in the program. 
Government insight teams and their industry 
counterparts should communicate on a regular 
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basis and continuously look for ways to 
improve implementation methods (El, E2, 
OD7). 

• Greater prime contractor involvement in 
technology development programs helps to 
ensure that the products developed have more 
immediate utility and require less rework (E3, 
E4). 

G. Technology Integration 

• Rigorous systems engineering and systems 
analysis processes should be utilized for 
identifying technical risk to accomplish 
requirements, technical performance metrics 
(TPMs), and figures of merit (FOMs) in the 
formulation stage. These should be used 
continuously to check progress against goals 
and objectives (Tl, T3, T5). 

• Programs should define, plan, and document 
the processes, procedures, and products for 
the SE&A function as early as possible in 
formulation (T4). 

• Programs should plan for and require 
adequate configuration control of analytical 
tools, models, and data sets and lock down die 
tool suite during analysis cycles (T6, T7). 

H. Organizational Design and Development 

• Dual project reporting paths cause confusion 
and introduce duplication of efforts. Clear 
lines of responsibility and authority are 
necessary for project success (OD1, OD2, 
OD4). 

• Execution of broadly scoped, long-range 
programs requires an environment of 
demonstrated trust coupled with appropriate 
delegation and empowerment to lower-level 
management (OD3). 

• Attention to die organization’s discipline, 
structure, and development is vital to the 
success of the mission. Opportunities to 
improve these at all levels should be 
encouraged (OD4, OD5, OD6, OD7). 

I. Program Integration & Communication 

• Integration is a key building block to an 
effective organization and should be 
performed on an on-going basis, especially at 
the middle-management level. A lack of 
integration, cited in the Columbia Accident 
Investigation Board (CAIB) Report as one of 


the causes of failure, is an issue in NASA’s 
culture. This lack of integration results in turf 
battles, “stove-pipes,” and chains of command 
issuing duplicate and confusing directives. 
These, in turn, adversely impact budget, 
schedule, and risk (PCI, PC 3, PCS, PC6, PC7, 
PC8). 

• Management should acknowledge employee 
concerns and have open, frank, and mutually 
respectful discussions. Such communication 
would allow for sharing “bad news” early and 
could shift the NASA culture to one of 
openness — with two-way. value-added, top- 
driven communication (PC2). 

• Reporting and readiness reviews should be 
streamlined. Projects should use standardized 
practices where practical (PC4, E5). 

J. Safety and Risk 

• Risks should be identified and tracked early 
and mitigation options incorporated into the 
acquisition strategy and systems analysis 
process (SRI , SR4, SR5). 

• A consistent, continuous risk management 
system should be utilized across the program 
and clear pass/fail criteria should be 
developed for risk reduction activities (SR2, 
SR3). 

ffl. Lessons Learned 
A. Requirements 

This section addresses requirements for products as 
derived from organizational vision and mission 
statements and/or from specific charters or directives 
issued by higher authorities. They may be refined 
through architecture studies, feasibility studies, and 
other concept exploration. 

1. Establish an Early Link Between Systems 

Engineering & Analysis and Technology 

Projects (Rl) 

Driving Events 

Several technology development activities from 2 nd 
Gen RLV and ASTP were combined to form the 
NGLT Vehicle Systems Research and Technology 
(VSR&T) Project. Initially, there was no strong link 
between the VSR&T Project and the SE&A projects. 
This link was needed to ensure that the proper 
architectures and, hence, technologies were being 
addressed. 
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Without the proper linkage to SE&A, the VSR&T 
Project may not have been addressing the critical 
technology requirements for the NGLT architectures 
under consideration. Later, the System Analysis 
Project (SAP) attempted to remedy the situation by 
interacting with the various NASA Centers 
conducting technology development. An 
Architectural Design and Technology Initialization 
Workshop was also held to bring together the various 
systems analysts and technology developers for 
NGLT, albeit after the fact. 

Lessons Learned 

Systems engineering and analysis should be linked to 
the technology development projects early in the 
program. Systems analysis should be used to develop 
quantifiable priorities and requirements against which 
technology projects should execute. The systems 
analysis activity should actively engage technology 
developers throughout the life cycle in developing 
requirements (working with hardware developers — 
e.g., utilizing the “Value Stream” process), ensuring 
models are valid, assessing the progress of the 
project, and ensuring the continued relevance of the 
investment against other, competing priorities. 

2. Develop a Set of Reference Mission 
Requirements Early (R2) 

Driving Events 

Due to the nature in which NGLT was formed, the 
SAP was forced to make certain assumptions 
regarding Program requirements in order to perform 
initial technology cost/benefit analyses. Once the first 
annual cycle started, requirements began to flow 
down from the Program, resulting in disconnects. 
This necessitated restarting the full cycle. 

Lessons Learned 

Programs should spend the time prior to each systems 
analysis cycle to develop and validate a reference set 
of mission requirements with customer, stakeholder, 
and analyst approval. 

3, Need for Early Reference Vehicle Conceptual 
Designs (R3) 

Driving Events 

In the 2nd Gen RLV Program, the prime contractors 
developing the vehicle architectures held their 
proprietary design information very close. 
Subsystem-level technology developers, e.g., those 
focused on propulsion and thermal protection 
systems (TPSs), were forced to derive a broad set of 


performance requirements due to the lack of a 
focused vehicle architecture. It was difficult for the 
subsystem technology developers to determine an 
appropriate set of environments and performance/life 
cycle requirements. Although NASA attempted to 
generate top-level requirements documents, a 
reference mission, and vehicle architecture, the 
information flow was slow and, in many cases, non- 
existent. In this case, the sub-element manager was 
forced to piece together design information from 
various prime contractors to arrive at a preliminary 
set of requirements encompassing all known vehicle 
architectures and environments. 

Lessons Learned 

A reference vehicle architecture, or clear 
performance specifications, should be in place prior 
to the initiation of technology development A clearly 
defined process is required for both generating and 
flowing down system requirements (e.g., associated 
flight environments) to the subsystem level. 

4. Plan Early for Incorporating Legacy 
Technology Programs (R4) 

Driving Events 

All NGLT activities were legacy projects from either 
the 2nd Gen RLV Program or the ASTP. While the 
legacy content represented strong candidates for any 
RLV application, it was not derived by applying 
consistent systems engineering practices (due to 
different parent programs). As a result, it was not 
possible to initiate a classical systems engineering 
process that would have driven out the needed 
technologies. This greatly complicated the NGLT 
systems engineering task. 

Lessons Learned 

The systems engineering team should be provided 
sufficient time to assess the cost/benefit of legacy 
projects and impacts of the new requirements on 
these projects. 

B. Plan 

This section is an articulation of the methods and 
systems to be employed for satisfying the 
requirements as well as their relationships. A plan 
typically combines considerations for technology, 
resources (e.g. budget, workforce, etc.), and schedule. 
As used herein, this includes the plan — both as a 
concept and document — as well as the actions 
required to develop the plan. Sections 3.3 and 3.4 
will address issues specific to budget and schedule. 
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1. Clearly Define Roles, Responsibilities, and a 
Common Set of Values (PI) 

Driving Events 

The Rocket Based Combined Cycle (RBCC) Project 
consisted of a consortium of prime contractors. 
Responsibilities were distributed throughout the 
consortium, but suppliers would only discuss issues 
with key decision makers, leaving the expediters and 
coordinators out of the information loop. In a multi- 
contractor consortium, the team eventually developed 
a set of core values: trust, teamwork, and critical 
knowledge; unfortunately, this came late in the 
program. A system that encourages team behaviors 
and discourages divisive behaviors should have been 
developed and matured earlier. 

Lessons Learned 

Programs should define roles and responsibilities 
early ail the way down to the supplier level with a 
clear focal point for communication. The 
government should play a key role in enabling the 
full team to communicate and enforce established 
protocols, responsibilities, and the team's core values 
early and often. This will provide expected standards 
of operation that are both established and fully 
understood for the life of the program. 

2. Plan for and Adequately Staff tbe Project 
Team (P2) 

Driving Events 

The X-43C Project was executed by the ASTP 
without performing a Non-advocate Review. This 
was based, in part, on similarity to the X-43A 
(Hyper-X) Project. The scope of effort was also not 
fully understood or supported by the executing 
Center management The first person assigned to the 
project was the Chief Engineer (CE), who also 
performed the Project Manager (PM) and Business 
Manager (BM) functions for nearly a year with 
minimal help from X-43A staff, who were working 
retum-to-flight activities. The CE was eventually 
promoted to PM and a new CE and a BM were 
assigned. Project staffing continued to lag needs, 
thereby forcing existing project staff to work long 
hours under stressful conditions. A lack of adequate 
workforce in the early days of the project delayed 
development of plans and complicated the project’s 
formulation. The X-43C’s similarity to X-43A did 
not reduce workload during formulation as assumed. 


Lessons Learned 

Each project should be treated as a new effort with 
adequate staff from the outset. A PM, CE, and BM 
should be assigned when a project is initiated. 
Technical leads for critical areas, a Deputy Project 
Manager, a Mission Assurance Manager, and Risk 
Manager should also be assigned as soon as possible. 

3. Allow Time for Adequate Project Formulation, 
Even When Task Appears Low Risk (P3) 

Driving Events 

Government cost estimates generated by the Booster 
and Launch Services (B&LS) Project were in the 
$48M to $52M range. Initial Rough Order of 
Magnitude (ROM) estimates from the contractor 
were 80% higher than the government estimates. The 
final proposal for this effort was approximately 58% 
higher than government estimates, despite the fact 
that government estimates were derived from X-43A 
actual costs and were considered to be quite accurate. 
Differences between the previous X-43A effort and 
the X^f3C effort were either unknown or 
underappreciated. Examples included contractor 
claims of hardware commonality and reusability, 
changes in governing range requirements, impacts 
due to both schedule and funding profile 
requirements, as well as increased technical and 
programmatic data requirements. 

Lessons Learned 

Do not underestimate the need for thorough project 
formulation and cost estimation, even when the 
project appears to be a straightforward follow-on 
effort. 

4. Thorough Preparation is Needed for Successful 
Acquisition Strategy Meetings (P4) 

Driving Events 

The X-43C Project initially planned to award a sole- 
source contract for the Demonstrator Vehicle (DV) 
based on similarity to the Hyper-X Program (X-43A). 
An unexpected level response to a sources-sought 
announcement revealed that a competitive 
procurement was feasible. Due to the size of the 
procurement, an Acquisition Strategy Meeting 
(ASM) was required. 
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A successful ASM briefing was developed by the PM 
and Contracting Officer. Briefing charts were 
developed in a multi-step approach. Two “realistic” 
rehearsals were held by the head of the langley 
Research Center (LaRC) Procurement with numerous 
questions for the presenters. After the charts were 
reworked and preparations were made for potential 
questions, a final briefing was completed. This level 
of preparation enabled the presenters to provide 
required information and perform well at the actual 
ASM, answering all questions and providing a well- 
rehearsed and smooth briefing. The ASM was 
conducted with excellent results and no action items 
were assigned. As requested, all subsequent actions 
were delegated to the LaRC Procurement Office. 

Lessons Learned 

Program should prepare well for ASMs, spending 
significant time rehearsing in realistic settings and 
answering questions from review teams. Programs 
should be prepared for the unexpected in major 
procurements by including Risk Based Acquisition 
Management (RB AM) in the Risk Management Plan. 

5. Provide Early, Comprehensive Test Facility 
Needs Definition (P5) 

Driving Events 

In an effort to reduce duplicity and cost, NASA has 
evolved and consolidated its test facilities over the 
past 10 years. This reduction in duplicity makes it 
unlikely that similar facilities will be available for 
“fly-offs” required under a competitive environment 
Additionally, test facility investment and sustainment 
is required to ensure that the facilities remain in a 
state of readiness to support program/project 
objectives. Relevant work in such facilities ensures 
retention of staff expertise for safety and for effective 
real-time management of the testing process. As an 
example, an independent assessment performed by 
the ARES Corporation determined that the test 
facilities available for RS-84 component and 
prototype testing were insufficient In addition, 
during 2003, NGLT unsuccessfully attempted to 
execute multiple test programs at the Stennis Space 
Cento’s (SSC’s) El Test Facility. These included the 
Integrated Powerhead Demonstrator’s (IPDs) 
Oxidizer Turbopump (OTP), Fuel Turbopump (FTP), 
RS-84' s Battleship Prebumer (BSPB), and TR-107’s 
Prebumer (PB)/Thrust Chamber Assembly (TCA). 
Ultimately, the TR-107’s PB/TCA testing was 
removed from the contract due to limitations in 
facility capability and capacity. 


Lessons Learned 

Limit test projects if required facility systems are not 
available. Avoid overly optimistic phasing of test 
facility projects and rely more heavily on historical 
capabilities. Allocate funding, as necessary, to allow 
for construction or modification of back-up facilities 
for mission /schedule-critical tests. During proposal 
evaluation, incorporate a process/procedure for 
evaluating the test requirements against known 
capabilities and capacities. Verify that existing and 
future test projects are considered along with built-in 
contingencies to reduce the schedule risks associated 
with overlapping tests. 

6. Assess Project Vendor Base Early (P6) 

Driving Events 

A contractor proposed to deliver a sub-scale 
prebumer for the RS-84 Project to SSC within 6 
months. The premise was its similarity to previously 
constructed hardware. However, as the design 
evolved, the similarities began to diminish, due 
primarily to differences in life and operating pressure 
requirements. A failure during the single-element 
testing also showed the need to make significant 
modifications to the original element design. Testing 
eventually began 9 months late, causing numerous 
schedule conflicts with other programs at SSC, as 
well as cost increases, which resulted in scope 
reductions. Delays in sub-scale main injector testing 
were estimated to be 12 months or more at project 
cancellation. Additionally, the ability of vendors to 
meet manufacturing schedules based on proposal 
estimates was greatly exaggerated. Schedule delays 
were caused by low production rates and by defects, 
which required rework of various parts. 

Lessons Learned 

Programs should perform a thorough assessment of a 
manufacturing vendor’s capability to produce space- 
qualified hardware and invest strategically to 
maintain necessary skills and capability for both 
near-term and far-term milestones. The government 
should critique proposals based on current 
requirements, not on past concepts with similar 
requirements. Programs should choose vendors 
based on current ability to perform, not solely on 
prior performance. 
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C. Schedule 

This section covers creating a nominal timetable for 
use of resources and achievement of result, tracking, 
and updating (i.e., planned versus actual), and the 
systems and teams involved in accomplishing those 
activities. 

1. Develop Detailed Schedules as Early as Possible 
(SI) 

Driving Events 

NGLT Systems Analysis 1 st Launch Vehicle Design 
Cycle. 

Lessons Learned 

Perform adequate planning and develop realistic 
schedules with adequate margins in coordination with 
the analysts and functional leads. Allow internal 
events to drive the schedule within the preplanned 
margins. Perform adequate assessment of the impact 
of external events and re-baseline the schedule if 
warranted. 

2. Develop an Integrated Master Schedule in the 
Formulation Phase (S2) 

Driving Events 

Due to the creation of NGLT as a merger of two 
different programs, the Program had not fully 
achieved one integrated master schedule linking all 
NGLT projects with standardized schedule formats at 
the time of Program closing. There was some 
difficulty identifying all key interrelationships 
between the projects. Another hindrance was the 
discrepancies with the Integrated Budget and 
Performance Document (IBPD) milestone dates and 
the milestone descriptions, especially during the 
merger process. Later, NGLT placed die milestones 
under full configuration management. 

Lessons Learned 

Programs should clearly identify milestones that all 
projects need to meet (versus a roll-up of project- 
level milestones) during formulation. This would 
allow the projects to produce schedules with direct 
links to a program integrated master schedule. The 
programs should require early compliance with 
schedule standardization guidelines, with some 
accommodations for contractor issues. 


3. Ensure Adequate Schedule/Staffing for 
Procurement Phase (S3) 

Driving Events 

During the X-43C procurement phase, the project 
failed to fully appreciate the timeline to execute 
respective procurement actions resulting in extended 
schedule. The full timeline duration from source 
evaluation board review to award was approximately 
one year. 

Lessons Learned 

The project should work with procurement staff to 
ensure that all issues have been addressed in 
developing a realistic procurement schedule, and then 
staff the procurement appropriately to ensure that the 
schedule can be met (right skill mix/number). 

4. Ensure Adequate Margin in Test Scheduling 
(S4) 

Driving Events 

ARES Corporation conducted an independent 
assessment at NASA’s request At the time of the 
independent assessment, it was determined that the 
RS-84 prototype phase duration and funding were 
likely to require either a schedule extension or the 
immediate start of fabrication of intermediate 
generations of major new components in order to 
complete component development testing, engine 
system testing, and possible hardware modifications 
due to design changes. Contingency cost and 
schedule to address several potential technical issues 
such as combustion instability, coking, ignition, 
durability, and material compatibility were not 
defined. 

Lessons Learned 

Formulation-phase schedules should include a 
realistic developmental test phase with margin that 
takes into account hardware repairs, facility 
maintenance, shift operations, re-engineering, and 
other real-world issues. 

D. Budget (B) 

Budget is, as noted above, part of the plan. It covers 
the location, tracking, and fiscal processing of 
resources. This categorization breaks “budget” out as 
a distinct element simply because it received so much 
attention. 
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1. Credible, Independent Cost Estimates are 
Needed Early in a Project (Bl) 

Driving Events 

The X-43C Project was a complex hypersonic flight 
demonstration. Three flight elements were linked to 
perform the mission. This scenario was similar to the 
Hyper-X (X-43A) Program, with the exception of the 
DV propulsion system. The X-43C Project 
developed an internal cost effort using X-43A actual 
cost data and heritage United States Air Force 
(USAF) engine cost data. Two independent cost 
estimates were performed that were within 10% of 
the project estimate. Contract proposal estimates 
came in more than 50% over these estimates. This 
created significant problems for NASA and the 
USAF, ultimately contributing to its cancellation. 

Lessons Learned 

The Agency should possess the ability to perform 
credible, independent cost estimates. This expertise 
could be externally procured from DoD or industry 
contractors. Major procurements should not be 
awarded before such a cost assessment is 
accomplished. NASA should implement multi-phase 
procurements, with options for major development in 
later phases and/or competitive awards of a separate 
conceptual design/cost development contract prior to 
competition for major hardware development 
contracts. 

2. Clearly Establish Reserves, Schedule Margin, 
and Spending Profile Early (B2) 

Driving Events 

The X-43C Project budget and milestones were 
established well ahead of detailed cost and schedule 
analysis. Per Office of Aerospace Technology 
(OAT) policy at that time, no budget reserves or 
schedule margins were allowed. As project cost and 
schedule requirements grew, the only choice was to 
slip the first flight The original spending profile was 
also not compatible with efficient procurement of 
flight hardware. Inability to adjust the spending 
profile forced the project to rearrange cost elements 
to match the profile, adding risk and extending the 
schedule. Lack of effective reserves, complicated by 
OAT 1-year spending metrics, exacerbated the 
problem. 

Lessons Learned 

Programs should establish preliminary project 
budgets with adequate reserves and schedule margin 
adjusting budget and spending profiles as better 


information is developed. Programs should allow 
rollover of unspent first-year reserve funds into the 
next year. The program should track and reclaim 
unspent reserves if they accumulate to excess. 

3. Centralized, Standardized Resources 
Integration is Critical to Success (B3) 

Driving Events 

Budget errors, omissions, and inconsistencies during 
formulation, coupled with the need for a focal point 
to respond to and coordinate with Headquarters 
Management and Center Chief Financial Office 
(CFO) personnel, were driving events. Complicating 
factors were introduced into the current resource 
environment as a result of the implementation of full- 
cost accounting and the IFM System. There was also 
a constant stream of questions and requests for 
budget information. 

Lessons Learned 

Programs should establish a central resources 
integration office and baseline the budget as quickly 
as possible, using a formal mechanism for tracking 
and approving changes (e.g., NGLT Program 
Requirements Control Board (PRCB). Detailed 
example: annotate budget spreadsheets with full 
configuration-control information, including a 
revision log). 

4. Diligence is Required to Prevent Unauthorized 
Expenditures Under the Current Integrated 
Financial Management System (B4) 

Driving Events 

Currently, it is possible to charge to a project code 
without a project manager’s or business manager’s 
approval. This has happened at least once on a 
particular project; it was not detected until it was too 
late to recover the funds. In cases like this, it is 
difficult to effectively manage the project budget and 
assess cost performance against the plan; at best, cost 
data are 4 weeks old. 

Lessons Learned 

Currently, IFM is not mature enough to effectively 
manage in-house project expenditures (e.g., no 
Earned Value Management (EVM) capability). 
Managers need a backup system until IFM evolves 
this capability. 
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5. Project Plans Should Accommodate “Real 
World” Funding Disbursements (B5) 

Driving Events 

Procurements and schedules have been significantly 
impacted by Continuing Resolutions (CRs) and other 
budget cycle issues. In addition. Centers often take a 
large percentage of (incremental) CR fund allotments 
for operating accounts, leaving a much smaller 
percentage for the project’s tasks, causing schedule 
slips due to the inability to fund contracts on- 
schedule. Problems are exacerbated by the 
uncertainties ami risks associated with advanced 
technology development 

Lessons Learned 

Project managers should develop a strategy for 
planning and scheduling that accounts for worst-case 
budget disbursement scenarios. Projects should also 
keep Headquarters offices informed of impacts such 
as Center use of CR allotments. 

E. Agreements (A) 

This very broad concept addresses the understandings 
reached so that resources can be allocated — or other 
actions taken — based on expectations of results and 
criteria for achieving those results. These agreements 
include contracts, grants, cooperative plans, 
articulation of a partnership, etc. 

1. Teaming Between Government Agencies for 
Early Technology Development Can Provide 
Significant Leverage (Al) 

Driving Events 

The development of the VSR&T Hypersonic 
Technology Experiment (HyTEx) Re-entry Testbed 
demonstrated that a properly constructed government 
partnership, which utilizes the most appropriate 
technical competencies and unique national assets, 
can provide a very efficient and unbiased approach 
for the demonstration and assessment of component 
technology performance. This model also minimizes 
duplication of effort and provides opportunities for 
leveraging, collaboration, and cost sharing where 
synergistic technology needs exist between various 
government organizations. With shrinking research 
and development funding available for technology 
development and maturation, significant benefits can 
be realized through pooling inter-Agency resources 
to increase leveraging opportunities between 
government organizations. The flexibility afforded by 
the use of existing government assets can allow for 
timely demonstrations of component technologies. 


allowing those technologies to be incorporated at a 
more mature level (reduced risk) earlier in the system 
development process. 

Lessons Learned 

A government teaming approach provides a viable 
option for the demonstration and assessment of 
component-technology performance. Specifically, in 
the case of a government-team approach, utilize each 
organization’s unique expertise, capabilities, and 
infrastructure (national assets); streamline the 
acquisition process by taking advantage of 
agreements in place between NASA and other 
government organizations and between various 
NASA Centers; allow a government-led team to act 
as a broker to coordinate component-technology 
demonstrations between appropriate test platform 
providers and the technology developers. 

2, Leverage Technology Expertise and Hardware 
from Other Government Programs (A2) 

Driving Events 

The Turbine Based Combined Cycle 
(TBCC)/Revolutionary Turbine Accelerator (RTA) 
project needed a high-performance turbine engine 
core. A new Centerline turbine engine development 
would cost at least $1B. The project was able to 
utilize existing assets from the DoD to provide the 
core of the turbine engine (i.e., YF-120 engine). The 
Project included DoD personnel on Integrated 
Technology Development (ITD) teams to leverage 
their expertise in turbine engine development, 
extensive experience with testing aggressive engine 
systems, and knowledge base of the YF-120 engine. 
It also included government technical experts from 
NASA and DoD in ITD teams to leverage their 
knowledge/involvement in base technology 
programs/projects such as Ultra-Efficient Engine 
Technology (UEET), Propulsion Research and 
Technology (PR&T), Integrated High-Performance 
Turbine Engine Technology (IHPTET) and Versatile, 
Affordable, Advanced Turbine Engine (VAATE). 

Lessons Learned 

Programs should canvas the government and leverage 
as much technology expertise, hardware, and 
facilities as possible from other programs to 
maximize return on NASA investments. 
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3. Partners Need Open, Honest Communications 
(A3) 

Driving Events 

The X-43C Project was a partnership between the 
United States Air Force Research Laboratory 
(USAFRL) and three NASA Centers. The Project’s 
management team aimed to utilize the strengths of 
each partner and minimize duplication of efforts to 
conserve resources. One NASA Center repeatedly 
tried to dramatically increase their management role 
in the program. Even though this Center appeared to 
accept their assigned role, they continually appealed 
to every level of authority in an attempt to have the 
Project structure altered. Eventually, an independent 
review of the project’s organization by the 
Independent Program Assessment Office (IP AO) 
validated the basic formulation of the project and its 
assignment of roles. 

Lessons Learned 

Select partners based on demonstrated strengths, 
matched with project needs. Construct partnership 
agreements with in-depth descriptions of how the 
partners will actually execute. (See lesson learned 
A2.) Hold face-to-face negotiations with all partners 
in attendance, so everyone benefits from the total of 
communications. Ensure that all participating 
partners understand and fully subscribe to their roles 
from the outset 

4. Complex Projects Require Detailed and 
Specific Agreements Between Agencies (A4) 

Driving Events 

X-43C began as a Joint NASA-USAF flight project 
involving three NASA Centos and the Air Force 
Research Laboratory (AFRL). The Project Manager 
inherited a relatively simple, brief Memorandum of 
Understanding (MOU) with the USAF that was based 
on a preliminary understanding of complexity and 
costs. USAF responsibility was for the development 
and procurement costs of the propulsion system. As 
contract costs for the propulsion system increased, 
the USAF was unwilling to increase funding above 
the level in the MOU. This forced the project to look 
for ways to reduce complexity and costs in a crisis 
management mode. OAT policies and differences in 
NASA and USAF culture made it impossible to 
indicate reserves in the MOU. This placed a severe 
strain on partnership relations when inevitable cost 
growth occurred and ultimately contributed to the 
Project’s cancellation. 


Lessons Learned 

Develop detailed Partnership Agreements that are 
based on sufficiently mature project formulation, or 
specify a definite schedule for revisions to 
incorporate details as understanding is achieved. 
Agreements should have the detail to ensure a true 
understanding of each party’s responsibilities. 
Specifically address responsibility for cost growth 
and indicate both budget and schedule reserves in the 
agreement. Sign agreements at levels high enough to 
ensure budget commitments can be met. 

5. Ensure Safety Requirements are Agreed to by 
Government Partners (A5) 

Driving Events 

On the IPD Project, a significant contract 
modification was made on a DoD- managed contract 
with MSFC serving as the technical lead. MSFC had 
no input on the modification and, as a result, 
hardware that failed to meet minimum NASA S&MA 
and accepted industry standards was accepted into the 
test article. Specifically, the hardware was “proof- 
tested” at only 80% maximum power level (with 
standard practice for pressure vessels being 125- 
150%) and no post-proof non-destructive evaluation 
was performed. As a partner with DoD on the 
subject project, MSFC reluctantly accepted hardware 
that failed to meet minimum Agency S&MA 
requirements. 

Lessons Learned 

In partnering agreements, all partners should be able 
to influence contract requirements and make changes 
as necessary. All partners should have the ability to 
impose additional S&MA requirements when 
deficiencies are discovered. 

6. International Relations Must Be Continually 
Nurtured (A6) 

Driving Events 

In the fall of 2002 and 2003, a few members of 
NGLT’s staff visited several countries in Europe to 
evaluate potential cooperative efforts. Upon these 
visits, the staff found that the cancellation of the X- 
38 in 2002 left very bitter feelings toward NASA by 
the European aerospace community. During these 
visits, the staff was able to initiate relationships with 
several companies and government organizations. 
Contact has continued at a very low level. 
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Lessons Learned 

NASA needs to continuously build international 
cooperation at the performing level. If NASA 
maintains a small effort, the Agency can ramp up 
quickly, as needed. 

7. International Agreements Process Within 
NASA Needs Streamlining (A7) 

Driving Events 

NGLT determined that it would be advantageous to 
partner with the Australian Centre for Hypersonics at 
the University of Queensland to obtain flight data for 
hypervelocity scramjet flow physics in a cost-sharing 
arrangement. The process, from initial discussion to 
approval from NASA Headquarters for a sole-source 
procurement involving a foreign entity, took 
approximately 15 months. Difficulties in budget 
phasing, technical progress, and communication 
resulted due to die duration of this process. (The 
effort was ultimately not pursued due to changes in 
program direction.) 

Enterprise-level approval was required for policy 
compliance. However, due to Center roles and 
responsibilities, approval of NGLT program 
management at MSFC, as well as line management at 
LaRC, was required. Guidance from Center and 
Headquarters export control officials was inconsistent 
at times, resulting in delays to interpret policy 
guidance and ensure approval of appropriate 
agreements. The Do D identified existing data 
exchange mechanisms that appeared to cover the 
scope of data exchange between the U.S. and 
Australia for this project However, NASA could not 
provide clear guidance on the use of these 
mechanisms or partnerships with DoD agencies that 
involve international cooperation. 

Lessons Learned 

Streamline the process by coordinating all 
international agreements and procurements (if 
necessary) through Headquarters functional offices, 
instead of the corresponding Center offices. 
Initiation of requirements should be made through 
program, and enterprise-level management. Engage 
appropriate Agency personnel to consider policy 
options (i.e., procurement, export control, legal) and 
identify areas of concern as early as possible. 
Maximize the use of existing project arrangements 
and data-exchange agreements (including those 
established through the DoD) to eliminate the need 
for NASA to pursue separate agreements if a suitable 
mechanism already exists. 


8. Industry Must Control Resources When 
Tasking a Government Laboratory (A8) 

Driving Events 

NASA Centers often desire to provide support (tasks) 
to industry-led proposals. This is typically 
accomplished via a Government Task Agreement 
(GTA). This agreement is essentially a subcontract to 
a NASA Center to perform work for a company. 
However, funding for this work does not come from 
or through the company to NASA. As a result ,the 
company does not have full management control over 
the NASA task. For example, a TPS technology 
developer requested a NASA Center to perform a 
series of tests. In fact, the Government facility did 
not have the test capability they claimed to have in 
the GTA and was unable to perform the testing. In 
another case, the Government facility did not perform 
the work agreed to in the GTA because other higher 
priority tasks bumped the work from the facility. 
Industry does not control the resources and, therefore, 
cannot redeploy to another test facility, thus 
endangering project success. 

Lessons Learned 

Avoid GTAs where the industry partner does not 
control the resources. Until other avenues are 
developed, industry should utilize other mechanisms, 
such as Space Act Agreements, to engage unique 
government laboratory capabilities. 

9* Streamline Contractor Data Deliverables (A9) 

Driving Events 

The original 2nd Gen RLV Program contract awards 
contained excessive contractor data deliverables that 
NASA did not have the resources to evaluate. The 
Airframe contract was renegotiated prior to the 
Option 2 award. At that time, the Contracting Office 
Technical Representative (COTR) generated a 
reduced set of data deliverables for the Northrop 
Grumman TA-2 program in order to lower overhead 
costs. Monthly reports became quarterly reports, and 
many documents were eliminated outright. The result 
was an increased number of engineering hours 
available to perform technical work within the budget 
constraints for Option 2. 

Lessons Learned 

Streamline data deliverables to prevent large 
contractor support labor costs; avoid inserting Data 
Requirements Documents (DRD) ‘‘boilerplates” into 
RFPs. Do not request more financial or technical 
data than the organization can actually analyze. 
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10. Standardize and Assure Earned- Value 
Clauses in Contracts are at the Lowest-Possible 
Level (A10) 

Driving Events 

Due to different (ASTP, 2 nd Gen RLV) contract 
clauses pertaining to business-related reporting 
requirements (deliverables), some prime contractors 
were not required to report EVM data and schedules 
at a low enough level to facilitate adequate schedule 
and budget analysis. Business data deliverables ware 
interpreted to be reportable only at the top Cost 
Performance Report (CPR) level, which is much too 
high to perform analysis or projections of problems 
and issues that may need to be addressed. 

Lessons Learned 

Assure clauses are incorporated into the contract that 
require contractors to provide electronic EVM data at 
the lowest planning level, as consistent with 
NASA/Program-level guidelines for level of EVM. 

11. Contract Statement of Work Should Address 
Critical Spares (All) 

Driving Events 

The failure of a pressure control valve during a test 
caused two pilot-operated relief valves to lift on the 
run tank, which was a driving event for the Auxiliary 
Propulsion Project (APP). A lack of spare hardware 
and soft goods resulted in several weeks of schedule 
impact that could have been mitigated. During 
testing of the Aerojet Reaction Control Engine 
(RCE), there have been several hardware failures. 
The lack of ready spares has significandy delayed the 
completion of testing. The Aerojet RCE is a new 
non-toxic, dual-thrust technology development with 
many unknowns. The spares philosophy should 
encompass manufacturing spares as well as test 
spares. 

Lessons Learned 

The statement of work (SOW) for new contracts 
should request a risk analysis of the 
testing/manufacturing planned for the project with 
critical hardware items identified and a spares plan 
included. The contract should have sufficient 
funding to initiate the spares procurement. Manage 
critical spares as project risk items. 


12. Contracts Should Allow Industry to Plan 
Future Work at an Adequate Level of Depth 
(A12) 

Driving Events 

An NGLT contractor not yet under contract requested 
a schedule change during one project phase that 
would impact another project phase. The 
government was unable to evaluate the change 
because the contractor was not allowed to produce a 
detailed schedule for future phases of the project. 

Lessons Learned 

Program should allow contractors to plan future work 
to a level of detail necessary to facilitate assessment 
of proposed changes. 

13. Tie Technology Contract Option Periods to 
Work Scope and Limit the Number of Options 
Per Year (A13) 

Driving Events 

As part of the 2nd Gen RLV Program, contracts were 
developed for technology development with plus 
additional option periods. The option periods were 
not based on the technical milestones but were based 
on program funding cycles and did not allow the 
contractor to optimize their SOW across the option 
periods. Long-lead procurements were a particular 
issue. As certain parts of the technology development 
incurred schedule slips, the project ended up with 
overlapping options so that work did not have to be 
stopped on other activities. At the end of the option 
period, the project was often not at a logical stopping 
point. If the next option period was not awarded 
NASA would have lost the return on the previous 
investment In some cases, NASA negotiated 
multiple options in the same fiscal year, diverting 
time and energy from the technology development 
task. 

Lessons Learned 

Programs should tie contract option periods to the 
technology development work plan, allow for long- 
lead procurements of effort needed for future options, 
and minimize the number of option periods to one per 
year. 
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14. Increase Time Between Option Award and 
Acceptance Test Plan (ATP) (A14) 

Driving Events 

Awarding tasks in options is problematic for 
industry. A 30-day period between a cancellation 
decision of an option and a new task inception did 
not allow sufficient time to re-deploy staff to other 
programs. Option “gates” created anxiety and 
uncertainty at the contractor level and disturbed the 
efficient flow of technical progress. 

Lessons Learned 

The Agency should increase the period between 
option award notification and the authority to proceed 
from 30 days to 60 days. 

F. Execution (E) 

This section includes ail lessons about 
implementation of the plans, the implemented, the 
associated sites and environments. These are lessons 
related to technology development and/or 
demonstration, especially where a specific 
technology is considered. 

1. Proper Government Insight Planning is 
Required (El) 

Driving Events 

Due to various internal and external drivers, recent 
space transportation programs have performed 
several cycles of program formulation, initiating and 
canceling projects prior to completion. Several 
models and management strategies have been 
employed related to teaming and contracting with 
industry. The RS-84 govemment/industry team 
developed an excellent insight relationship that 
should be considered by other projects. Recent 
experience within NASA has revealed die need to 
maintain adequate skill and insight levels for 
complex space initiatives. The CAIB Report cited 
lack of adequate technical insight for critical safety 
issues and recent development experiences such as 
X-33 and X-37 also reveal a lack of government 
involvement in key assessments such as technology 
readiness and design margins. 

Lessons Learned 

NASA should provide technical experts to review 
contractor data deliverables and provide independent 
analysis in critical, high-risk areas. Projects should 
decide early on the level of insight required and 
should allocate resources for that work. Government 


insight teams and their industry counterparts should 
communicate on a regular basis. The government 
should not relieve the contractor of responsibility for 
success. However, contracts must account for the 
role of government, and the contractor should be 
rewarded for working with insight teams. 

2. Independent Analysis of Critical Items can 
Save Money (E2) 

Driving Events 

The IPD Project conducted liquid-oxygen 
powerpack-test operations that required that detailed 
test requests be generated, processed, and approved 
by multiple entities. The test request was generated 
by the contractor, converted to the test area format, 
reviewed by the contractor and government 
personnel, and then released to the test operations 
personnel to execute. On two separate occasions, this 
process was followed and the request was released to 
test operations personnel, only to be found in error. 
Because of the criticality of the request (i.e., IPD had 
only a single hardware unit) an independent civil 
service review of the test request was performed. 
When the test request was compared to the predicted 
test profiles provided by the contractor, performance 
redline errors were discovered that would have 
resulted in premature cut-off of a good test. Catching 
these errors saved the project the cost associated with 
conducting a second test and subsequent test 
turnarounds (approximately $100K apiece). This 
check avoided cost overruns and schedule delays; 
however, errors in other performance values could 
have resulted in a loss of test articles. 

Lessons Learned 

Programs should ensure there is an independent 
review of mission-critical functions before executing 
them. 

3. Prime Contractor Should Be Responsible for 
Inspections (E3) 

Driving Events 

The IPD Project utilized contracts based on Air Force 
standards with prime contractors. For a research and 
development system, this allowed the contractors to 
define inspections and quality control requirements. 
In a cost-saving measure, the engine integrator 
contractor chose to delegate to their subcontractors 
the responsibility for performing final inspections on 
hardware they fabricated prior to shipment. Much of 
the delivered hardware configuration was correct; 
however, there were several pieces that were not 
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manufactured to print, which impacted the engine 
assembly schedule. Similar problems occurred during 
duct, turbopump, and prebumer fabrication. The root 
cause of most of these issues was subcontractor 
methods of operation. If the final inspection process 
was performed by the prime (or possibly by an 
independent inspection group), it is likely that these 
hardware discrepancies would have been detected 
prior to installation (at which point the greatest 
schedule impact is experienced). The prime 
contractors appeared to have higher quality 
inspection standards and capabilities than many of 
the subcontractors. 

Lessons Learned 

In contracts involving hardware manufacturing or 
processing, NASA programs should stipulate that 
primary and final inspection responsibility be 
maintained at the prime contractor level. The prime 
should perform as many of the required inspections 
as possible. 

4. Involve Industry Early in Research and 
Technology Activities (E4) 

Driving Events 

In the 2 nd Gen RLV Program task, “Ceramic Matrix 
Composite Control Surface Technology 
Development,” the in-house/ vendor team elected to 
use a processing approach that reduced processing 
risk by fabricating multiple, separate components, 
subsequently assembling them using fasteners. Much 
success was realized using this approach: a 

Carbon/Silicon Carbide (C/SiC) structural element 
and a half-scale C/SiC subcomponent were 
successfully fabricated and tested under extreme 
environments never before experienced by any hot 
structure component. However, through participation 
in the X-37 hot structure control surface program, the 
project learned that the separately processed and 
bolted assembly approach cannot meet tight 
manufacturing tolerances required for flight control 
surfaces. 

Lessons Learned 

Increasing the involvement of knowledgeable 
personnel from aerospace prime contractors in 
technology development programs ensures that 
products will have greater utility to aerospace 
applications. 


5, Streamlined Design Review Processes can be 
more Effective (E5) 

Driving Events 

The VSR&T Project HyTEx Re-entry Testbed 
Preliminary Design Review (PDR) process was 
performed using concurrent engineering techniques 
to significantly reduce the overall design review 
cycle time and travel requirements without 
compromising the comprehensive breadth of the 
design review process. The HyTEx government 
partnership took advantage of best design review 
practices followed by the various NASA Centers, 
DoD, and the Department of Energy (DoE) to derive 
a streamlined design review process. 

Lessons Learned 

A streamlined design review process can save time 
and money, while maintaining an appropriate depth 
in the review process. NASA programs should carry 
out proper planning prior to the review and utilize 
fully engaged participants. Some specific 
recommendations: 

1. Define design review objectives, exit criteria, 
and process criteria to convene the Board early 
on; 

2. Management buy-in on process to be followed 
is critical early on; 

3. Identify reviewers, screening teams, review 
teams, pre-board members, and board 
members that are willing to engage in the 
process; 

4. Require electronic server-based tools to 
capture design review documentation and to 
support the Review Item Discrepancy (RID) 
process; 

5. Provide review data packages prior to formal 
presentations with a short pre-review period 
where pre-RIDs are accepted (with time 
needed to review before presentation); 

6. Design review presentations are limited in 
duration with a hard cut-off for RID 
submission at the end of the presentation 
process; this requires reviewers and 
screening/review teams to be present and 
engaged; 

7. Screening teams assign RIDs to the 
appropriate review teams at the end of each 
day’s presentation; 
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8. Review teams disposition present RIDs for 
pre-Board actions following formal review 
presentations (i.e., do not solve the problem 
but assign the appropriate action); 

9. Pre-Board makes recommendation whether or 
not to convene the Board based on pre- 
determined criteria; and 

10. Management out-briefs of the review process 
and results should be planned and scheduled 
up front. 

6. Automated Tools Cannot Replace Experienced 
Analysts (E6) 

Driving Events 

In the NGLT SAP, the automation of life-cycle tools 
was attempted, but the automation was not 
sufficiently flexible to eliminate the need for 
experienced expert human analysts. 

Lessons Learned 

Programs should not rely solely on automated tools. 
Human experts should remain in the loop. 

G. Technology Integration (T) 

This theme addresses SE&A as applied to provide 
feedback and adjustments within “program-phase 
themes” (i.e., “requirements” through “execution”) 
regarding considerations for technology per se. For 
example, this covers the formulation, measurement 
and analysis of TPMs as well as modifications — due 
to TPM-based feedback — to plans and agreements. 

1. Rigorous Systems Engineering is Needed in 
Technology Projects (Tl) 

Driving Events 

It was noted by the TBCC Project that timely detailed 
systems analyses/engineering was needed within 
projects to make effective cost/benefit decisions 
regarding configurations and technologies. For 
example: early in the TBCC Project, legacy hardware 
(HiMATE augmentor rig) was proposed for flame 
stability testing. However, cost benefit analysis, 
systems analysis, and leak testing of hardware 
showed that new rigs (flame stability and 
annular/sector rigs) would be technically beneficial 
and financially prudent The RS-84 Project identified 
a single-shaft turbopump as the baseline at the time 
the proposal was negotiated. Over time, it was 
discovered that a single shaft for the turbopumps 
would require significantly more risk management 


than previously identified. The systems engineering 
process allowed this to be incorporated into the 
program decision-making process. During the 
Northrop Grumman Reaction Control System (RCS) 
Liquid Oxygen (LOX)ZEthanol testing, the project 
experienced three occurrences of a failure of the 
disilicide coating on the C103 chamber. The project 
team was under significant pressure due to schedule 
delays, so an attempt was made to apply a quick fix 
of the problem without fully understanding the 
issues. However, the Project Office required the 
contractor to perform a full fault tree analysis of the 
problem before attempting any solutions. The team 
subsequently determined that a new Platinum-Iridium 
(Pt-Ir) chamber, rather than the coated C103 
chamber, could be used to meet the test objectives 
without having to solve a coating problem not in the 
task objectives. 

Lessons Learned 

SE&A should be executed within technology projects 
to ensure timely decisions (which are supported with 
an appropriate depth of analysis e.g., cost/benefit) 
and coordinated with top-down program-level 
systems engineering and analysis/requirements flow- 
down. Programs should ensure the systems 
engineering discipline is applied even when stress is 
high and time is short. 

2. A Rapid Response Systems Analysis Team is 
Invaluable (T2) 

Driving Events 

Routinely, questions (many hypothetical) from 
Program stakeholders arose during Program 
progression. Answering some of these questions 
called for analysis of various concepts and 
architectures that were not previously identified. 

Lessons Learned 

Programs should set aside a quick response “reserve” 
of resources (manpower and budget) to perform mid- 
course checks and respond to senior management 
“what if’ inquiries as necessary. 

3. Effective Use of the “Value Stream” Method 
Requires Active Engineering Organization 
Participation (T3) 

Driving Events 

The Program incorporated “Value Stream” as a 
method to link goals and objectives to tasks being 
performed. Because of competing priorities. Value 
Stream often did not have the right level of 
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engineering organization participation adversely 
affecting the results. 

Lessons Learned 

Programs should ensure performing engineering 
organizations are tasked and funded to support 
activities as a part of on-going design activities, 
rather than as “add-ons,” 

4. Early Development of a Systems Engineering 
Management Plan Is Critical (T4) 

Driving Events 

NGLT experienced continual disagreement over 
processes, procedures and products to be used by the 
Program’s SE&A teams, all of which came from 
different legacy programs, each possessing different 
requirements and Center perspectives (e.g., esearch 
versus development). For example, inconsistencies in 
weights and sizing models and differences in 
definitions of subsystems, combined with insufficient 
technical exchange with the various architectural 
teams, led to incorrect interpretation of data, and 
inaccurate and/or misleading results that required 
rework. 

Lessons Learned 

Programs should define, plan, and agree to processes, 
procedures, and products for the SE&A process as 
early as possible and formally document in a SEMP. 

5. Establish and Flow-down Technical 
Performance Metrics (T5) 

Driving Events 

ARES Corporation completed an independent 
assessment of the IPD Project at the request of 
NASA. At the time of the independent assessment, it 
was determined that IPD Project TPMs (e.g., achieve 
250-KLb thrust, to throttle between 50% and 100%, 
etc.) were not appropriate for project objectives, i.e. 
development of a start sequence and validation of key 
components and tools. Unresolved component issues 
(such as oxygen-turbopump lift-off seal, fuel 
prebumer-combustion instability and oxygen PB 
ignition) threatened the success of system tests. At 
the time of the independent assessment, it was 
determined that NGLT Level 1 TPMs and goals had 
not been formally translated to Level 2 system TPMs 
or Level 3 component TPMs. As a result, the 
technologists had generated Level 3 TPMs without 
input from the Systems Integration Project (SIP)/SAP 
on Level 1 goals. 


Lessons Learned 

Programs should develop trackable, requirements- 
based TPMs that will help manage the project 
performance based on stated objectives. 

6. Provide Configuration Control of Models, 
Methods, and Data Sets (T6) 

Driving Events 

Many NGLT analyses had several iterative runs that 
were captured only by the analyst performing the 
analysis. The acceptance of this “tribal knowledge” 
capture is something that costs time and money in the 
future by driving the need to “reinvent” analyses to 
validate the touted solutions. 

Lessons Learned 

Adequate documentation must be coupled with 
configuration control on analytical tools, models, and 
data sets actually applied. 

7. Freeze Methods/Tools During the Analysis 
Cycle (T7) 

Driving Events 

In performing simultaneous tools development and 
systems analysis, it was discovered that analysis 
activities must be insulated from on-going tool 
development activities and that tools configuration, 
and methodologies must be frozen during each design 
cycle. 

Lessons Learned 

Programs should separate tool development activities 
from analysis activities and freeze the methodologies 
and tools prior to initiation of each design cycle. 

8. Space Transportation Life Cycle Analysis 
Tools Need an Overhaul (T8) 

Driving Events 

It was realized early in the NGLT SAP that the 
analysis tools and underlying databases for life-cycle 
analysis needed an urgent upgrade. To address this 
need, SAP created a team called the Life Cycle 
Analysis Team (LCAT) to develop methodologies to 
improve the analysis of reliability, supportability, 
development cost, recurring cost, safety, availability 
and other life cycle-related parameters. One example 
of needed improvement is the area of operations 
costs. Current tools do not allow for the evaluation of 
creative new methods of performing operations. All 
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current tools are based on Expendable Launch 
Vehicles (ELVs) and Shuttle; accurate cost cannot be 
developed for operations designs that vary 
significantly from these operations approaches. One 
method would be to develop a discrete event 
simulation to model the operations costs and allow 
the program to represent costs for more creative 
operations approaches more accurately. 

Lessons Learned 

Continued investment is needed in life cycle analysis 
tools, methods, and databases. During the 
development of systems, it is essential to understand 
impacts on the life- cycle costs, safety, reliability, etc. 

9, Utility Curve Analysis Method Aids Complex 
System Trades (T9) 

Driving Events 

To design a new rocket engine for the NGLT 
Program, numerous trade studies had to be 
conducted. In each trade study, a choice had to be 
made between competing designs to select the 
optimal solution for the engine. Potential design 
options also had to be evaluated against multiple 
program objectives, some of which conflicted at 
times. The prime contractor for RS-84 implemented a 
utility curve analysis to help drive decisions during 
difficult, complex design trade studies. The method 
involves polling or surveying of the applicable 
customer base to allow numerical determination of 
utility indifference curves. The customer preferences 
toward program goals such as low-cost, high- 
reliability, high-thrust to weight ratio, etc., were then 
weighted appropriately. The utility analysis allowed 
the determination of a numerical score for each 
potential design solution. The design option with the 
highest score best met the customer objectives and 
was selected for the final engine design. 

Lessons Learned 

A utility curve analysis approach to assessing 
detailed trade studies should be considered in future 
activities, especially where numerous design 
solutions exist in an environment with multiple 
program goals and objectives. 

H. Organizational Design and Development (OD) 

This section addresses the formal and informal 
organizational structures and element-to-element 
relationships. Roles and responsibilities for 
individuals and groups are key here. Emphasis is on 
changing to meet changing needs and situations. 


1- A Simple, Clear Management Chain is Critical 
(OD1) 

Driving Events 

The X-43C Project was required to report up two 
management chains to the Center level, one for 
‘‘programmatic” issues and one for “implementation” 
issues. These areas of a project are not separable, as 
they continually interact. In addition, there was also 
a direct management chain to the Headquarters-level, 
NGLT Program Management Team. Often, an action 
would be levied on the project from two or more 
sources, with slightly different interpretations, 
disguising the fact that it was a single action. 
Management issues surfaced that were addressed 
through one chain, often offending the other. One 
chain sometimes challenged the other in overlapping 
areas, causing confusion for the project. 

Lessons Learned 

Projects should report through a single, clear Center 
management chain. Programs should minimize the 
number of levels between the Project Manager and 
the Center Director for significant and/or highly 
visible projects. A single Center-level manager 
should be responsible for making final decisions at 
that Center for a particular set of projects. 

2. Project Plans Must Clearly Specify Roles and 
Responsibilities (OD2) 

Driving Events 

Organizationally, the Booster and launch Services 
(B&LS) Project Manager was at the same level as the 
X-43C DV Contract Manager. The Level HI Project 
Office possessed all technical and managerial 
expertise required to execute the project scope. The 
DV Contract Manager did not have budget and 
schedule authority. DV technical insight was 
provided by the Level II technical staff. Difficulties 
arose as the Level II staff attempted to exert the same 
influence on the B&LS Project as it had on the DV 
Contract. A Level II Project Plan that fully explained 
the project roles and responsibilities and clarified 
project controls was not approved prior to project 
cancellation. In contrast, the NGLT SAP 
management established and documented clear goals 
and individual responsibilities at the beginning of the 
project formulation, so that all team members 
understood what was expected of them. 

Lessons Learned 

Programs and Pprojects should write a Project or 
Program Plan early, leaving some place-holder 
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sections in order to establish the control mechanisms 
that will be employed on the project. Early releases 
of the plan can be baselined at the project- or 
program-level only. Specify how critical financial, 
schedule, and technical decisions will be made. 
Controls (i.e., approval gates and processes) should 
be clearly defined in concert with the level of 
responsibility delegated to the project or sub-project. 

3. Delegate Responsibility and Then Follow 
Through (OD3) 

Driving Events 

The ASTP (and later NGLT) Program delegated 
much responsibility for the X-43C Project to LaRC. 
A single Project Manager was given authority to 
manage the project for the parent programs. The 
Program Manager provided general guidance without 
interfering with project management. Appropriate 
status reporting was required of all projects by the 
Program. This arrangement applied a sound 
management approach and demonstrated trust in the 
Project Manager and his Center managers. In effect, 
the Program Manager defined and enforced a chain 
of command that fully supported the Project 
Manager’s authority. 

Lessons Learned 

1 . Program Managers should delegate 

responsibility for project execution, with 
matching authority, to the subordinate Project 
Managers. Enforcement of this chain of 
command facilitates effective project 

management at supporting Centers. 

2. All projects should have a single manager 
with appropriate authority to execute the 
project 

3. Status reporting to the Program should be 
enforced at the lowest level of detail and 
frequency, consistent with project complexity 
and visibility. 

4. Identify a Single Leader and Property Staff 
Systems Engineering Activities (OD4) 

Driving Events 

The NGLT Systems Engineering Office was not fully 
capable of meeting expectations due to inadequate 
staffing, unclear roles/responsibilities, and inadequate 
planning. Multiple organizations performed systems 
engineering functions across multiple Centers with 
unclear lines of authority. 


Lessons Learned 

Programs should have a single authority for systems 
engineering and integration, obtaining clear 
commitments from participating Centers on both 
lines of authority and the deliverable work package. 
Related staffing issues should also be resolved up 
front 

5. Encourage Management and Organizational 
Development (OD5) 

Driving Events 

NGLT contracted with professional organizational 
consultants to facilitate the development of 
management skills within the organization. 
Workshops focused on developing understanding of 
individual roles as well as roles of those above, 
below, and beside them. This basic understanding led 
to increased effectiveness at all levels of the 
organization. 

Lessons Learned 

NASA programs should provide regular 
organizational development opportunities to all 
Program personnel. 

6. Establish Formal Change Control at All Levels 
(OD6) 

Driving Events 

A formal change control process was not in place at 
the X-43 Project level (Level II). The Level III 
project incorporated DV design requirements such as 
outer mold line, mass properties, and separation point 
requirements into its Launch System Requirements 
Document (LSRD). Over the life of the project, 
changes were made in the DV baseline by the Level 
II office without official notification to the B&LS 
Project (Level HI). 

Lessons Learned 

NASA programs should establish a formal change 
control process and critical baselines early at all 
levels of an activity (Levels I, n, HI), and ensure that 
these baselines are communicated clearly across all 
elements. 
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7. NASA’s Unique Organizational Perspective 
Can Help in Problem Identification and 
Resolution (OD7) 

Driving Events 

The IPD Project was the first project to test a liquid 
oxygen turbopump fully supported by hydrostatic 
bearings. This unique bearing design benefits 
include ‘"floating” the rotating shaft on a thin film of 
liquid oxygen fluid. Because NASA technical 
experts were not part of the design team (being 
designed when IPD was an Air Force-only project), a 
detailed technical review was held prior to the initial 
test. During this review, concern over electrostatic 
discharge was raised. The contractor responded that 
the phenomena had not been encountered in liquid 
oxygen before and therefore was not a concern. 
NASA reviewers were aware of similar issues with 
other contractors and from in-house experience. 
NASA personnel disassembled a similar in-house 
turbopump looking explicitly for evidence of 
electrostatic discharge. Evidence showed that the 
phenomena had occurred in liquid oxygen, and 
indicated that the severity of die arcing was sufficient 
to ignite the carbon lift-off seal in the oxygen 
turbopump should the arc occur at that seal, with 
potential catastrophic results. 

Lessons Learned 

Critical aspects of contractor designs and tests should 
be reviewed by NASA subject matter experts to 
increase the likelihood of success. 

L Program Integration and Communication (PC) 

This section covers the relationships and interactions 
between program elements — by formal or informal 
mechanisms — that correlate to achieving the greater 
good of the program objectives/goals. Transmittal of 
information — communication — is an essential 

precursor to any decision or action. Hence, 
communication is an integral part of successful 
program integration in its broadest sense. 

1. Middle-Management Integration Is a Powerful 
Tool (PCI) 

Driving Events 

The NGLT middle-managers forum for integration 
across the program greatly improved overall program 
execution. This forum was championed by one 
middle-manager who knew that his peers would have 
a tendency to pull away from each other and work in 
their own “silo.” This would have caused a lack of 
integration across the program and likely less 


teamwork among the program office staff. The forum 
set-up caused these middle-managers to take 
integration seriously and avoided costly breakdowns 
in communication at their level. 

Lessons Learned 

NASA programs should create a middle-managers’ 
forum for integration at the Program Office-level. 
Program managers should pay close attention to how 
middle managers integrate with their peers and 
ensure communication is flowing, thus contributing 
to improving safety, eliminating duplication, and 
promoting good working relationships across the 
elements of the Program/Centers. Programs should 
also explore the potential for applying this concept at 
other levels of the organization. 

2. Create an Environment That Will Accept “Bad 
News” by Management (PC2) 

Driving Events 

A program’s accomplishments depend on good, 
open, and integrated communication. The CAIB 
Report highlighted the issue of keeping a “code of 
silence” and its detriment to the success of a 
program. This occurred within NGLT as well. One 
example was how the SE&A function’s project-level 
management had issues in working with each other 
and the Program Office; as hard as they tried, they 
could not be as open as they needed to be to resolve 
their issues. 

Lessons Learned 

Programs should consider the “Crucial 
Conversations” approach used in NGLT, which 
effectively deals with organizational cultures that 
historically encourage silence or retribution. 
Educate/train people at all levels of the organization 
and give them a set of tools that will help in 
overcoming this detrimental way of operating. 

3. Document Decisions and Supporting Rationale 
(PC3) 

Driving Events 

In some instances, there were not enough details 
regarding the decision rationale for technology and/or 
launch vehicle analysis for validation of analytical 
results to prevent duplication of effort. 
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Lessons Learned 

NASA programs should provide adequate 
documentation of decision rationale for validation of 
analytical results to prevent later duplication of 
effort. 

4. Standardize and Minimize the Number of 

Different Project Reporting Mechanisms (PC4) 

Driving Events 

The X-43C Project reported progress to ASTP (later 
NGLT) first on a monthly basis and later in a 
Quarterly Report, each with a unique required chart 
format. During the same timeframe, X-43C reported 
to the LaRC Center Program Management Council, 
which required yet another chart format. All chart 
sets contained largely the same project management 
data and status reporting. However, the formats were 
different, requiring the project team to create multiple 
versions of the same data. This increased the 
workload and added risk of error due to the rework 
process. 

Lessons Learned 

Programs should adopt a common format for project 
management reporting and coordinate with 
Headquarters and each Cento's Systems 
Management Office to enhance communication and 
reduce workload on project staff. Programs could 
also develop a “super-set” of formats to cover 
anticipated reporting needs at all levels. 

5. Project Teams and Systems Analysts Need to 
Work Closely When Performing Technology 
Assessments (PCS) 

Driving Events 

Due to the nature in which NGLT was created, there 
was not initially a tight integration between SE&A 
and the technology projects. Technology projects 
were not cognizant of what was going on within 
SE&A activities. In early FY04, the SE&A team held 
a “road show” that took the results of the previous 
cycle of analysis to all Centers to demonstrate their 
activities to the technology projects. This team also 
conducted an Architecture Design and Technology 
Integration Workshop prior to the start of the next 
analysis cycle to discuss the technology impacts on 
system designs and identify a viable baseline and 
alternate technology options. This workshop 
provided a forum for technologists to discuss how 
their technologies impacted system designs with the 
systems analysts. In the past. Value Stream 
workshops were held at the end of each cycle to 


facilitate a meeting between technologists and 
systems analysts to identify shortfalls in capability 
versus need. 

Lessons Learned 

Technology projects should be made aware of 
systems analyses that are being performed and the 
parameters that are being used throughout the 
assessment process. The use of structured forums at 
the beginning of the analysis cycle and a Value 
Stream workshop at the end of each cycle should be a 
part of any technology program. Representatives 
from each technology project should be represented 
on the systems analysis team(s). 

6. Set Clear Expectations in Design Reviews 
(PC6) 

Driving Events 

At a recent PDR, it became obvious to the 
Government team that the contractor did not 
understand NASA’s expectations and requirements 
for the review. 

Lessons Learned 

NASA should ensure that expectations and 
requirements for contract items such as milestones 
and deliverables are clearly stated in the contract 
SOW and understood by the contractor. 

7. Effective Vertical Communications are a 
Necessity (PC7) 

Driving Events 

During the life of the RS-84 Project, several team 
members expressed frustration with their lack of 
knowledge regarding activities at the project, 
program, and Headquarters levels. Team members 
felt that there was inadequate ‘"top-down” 
communication. 

Lessons Learned 

NASA programs should consider interactive 
communications rather than the one-way variety such 
as e-mail. Senior program personnel should schedule 
times to conduct project team “all-hands” to speak to 
team members and answer questions. 
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8. Capture “Lessons Learned” Throughout the 
Project Lifecycle (PC8) 

Driving Events 

During X-43C Project development, many lessons 
learned from the X-43A Project were incorporated 
informally by shared technical and management 
personnel. Because of shared resources, many 
management and technical decisions and subsequent 
activities were not formally captured. This situation 
drove the need to repeat communications regarding 
decision rationale to new personnel and outside 
reviewers. 

Lessons Learned 

Lessons learned activities should be performed at the 
working level on a continuous basis throughout the 
life of a project, as are risk management reviews. 
Lessons learned activities should be tied to risk 
management activities because they often result from 
unidentified, unmanaged, or misunderstood risks. 

J. Safety and Risk (SR) 

Safety and risk are affected, often to a first order, by 
changes anywhere within the Program. Lessons 
learned in this section are limited in number, since 
safety is best appreciated as being “everywhere.” 
Additionally, managers of advanced-technology- 
development projects are increasingly using risk 
management as a primary tool to guide most aspects 
of their projects. 

1. Identify and Track Risks Early and 

Incorporate into Acquisition Strategy (SRI) 

Driving Events 

The X-43C Project compiled and managed risk early 
in the project formulation. By initiating the risk 
process early on, project plans and associated 
reasoning could be explained early and effectively. 

Lessons Learned 

NASA programs should prepare risk management 
plans early in project formulation and include budget 
analysis and Risk-based Acqusition management 
(RBAM) activities to serve the project during 
formulation and startup. 


2. Use a Consistent Risk Management System 
Across the Program (SR2) 

Driving Events 

NGLT technology content represented risk mitigation 
activities for RLV. Risk assessment and management 
were the responsibility of each project team, each of 
which brought a different system from their legacy 
programs. The most commonly used system was 
resident in the Space Transportation Information 
Network (STIN), but some projects used other 
Commercial Off-the-Shelf (COTS) tools. While risk 
management was performed, it was not performed at 
a uniform level across the Program. The STIN risk 
tools were under constant development, which also 
complicated matters. 

Lessons Learned 

Programs should select a proven risk management 
system that represents an industry standard and then 
train the users in its application, making sure that 
expectations are understood from top to bottom in the 
organizations that are to be using the system. 
Programs should also provide mandatory risk 
management training to all members of a 
program/project team (including contractors) with 
periodic refresher training to accommodate personnel 
changes. 

3. Develop Specific Pass/Fail Criteria for Risk 
Reduction Activities (SR3) 

Driving Events 

The RS-84 Risk Management Program evolved as the 
Rocket Engine Prototype (REP) Project matured. 
Each IPT conducted monthly meetings, with NASA 
participation, during which a status of each risk was 
discussed. Descriptions of risk mitigation/reduction 
steps were updated and refined, and actions were 
assigned for implementation of the plans. New 
candidate risks were presented for review/validation 
and, if approved, new risks were assigned owners for 
assessment and planning. 

Lessons Learned 

Programs should ensure pass/fail criteria are 
established as an integral part of the planning phase 
for risk reduction and mitigation activities. Programs 
should develop specific decision criteria for 
implementation of back-up/contingency plans or 
modification of the design or development effort. 
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4. Perform Technology Risk Assessment as an 
Integral Part of Systems Analysis Efforts (SR4) 

Driving Events 

A technology risk assessment was performed for 
NGLT on an uncertain schedule, often hurriedly at 
the end of an analysis cycle. In some cases, there 
was inadequate time to fully validate the results of 
the assessments. 

Lessons Learned 

Programs should plan for continual interaction 
between the end-users and the design analysts, such 
that performance of die risk assessment process will 
provide the desired benefit. 

5. Develop Alternate Strategies when 
Incorporating New Materials and Processes 
(SR5) 

Driving Events 

ARES Corporation completed an independent 
assessment of the IPD Project at the request of 
NASA. The following lessons learned were part of 
the resulting findings. 

Lessons Learned 

Projects should develop and demonstrate 
qualification processes for flight hardware fabrication 
and inspection during the prototype design and 
fabrication phase and develop proven, fall-back 
approaches (e.g., heavy welded ducts, oxidation 
resistant coatings, etc.) to control expenditures if 
fabrication or testing difficulties arise on prototype 
components. 
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